Skip to content

wtclient: prune closable sessions when the tower is unreachable - #10947

Open
jkuchar wants to merge 2 commits into
lightningnetwork:masterfrom
jkuchar:fix-wtclient-dead-tower-cleanup
Open

wtclient: prune closable sessions when the tower is unreachable#10947
jkuchar wants to merge 2 commits into
lightningnetwork:masterfrom
jkuchar:fix-wtclient-dead-tower-cleanup

Conversation

@jkuchar

@jkuchar jkuchar commented Jul 1, 2026

Copy link
Copy Markdown

Fixes #10646

Problem

When every channel backed up by a watchtower session is closed, the session becomes "closable" and handleClosableSessions tries to notify the tower that it may delete the session (via deleteSessionFromTower) before removing the client's own copy.

If the tower is permanently unreachable (offline, or deactivated/removed by the user), that dial fails, the loop continues, and DB.DeleteSession is never reached. As a result:

  • The closable session is never removed from the client DB, so on every restart ListClosableSessions reloads it and the client re-dials the dead tower. With many sessions this keeps lnd busy for a very long time on startup (the reporter saw "hours"), emitting a stream of error deleting session ... from tower: failed to dial tower(...).
  • The tower can never be fully removed, because wtdb.RemoveTower only deletes a tower that has zero sessions — otherwise it just marks it inactive. So the dead tower's sessions, and the tower itself, are stuck forever (also the subject of [feature]: cleanup of wtclient.db / removal of watchtower #7035).

Fix

Notifying the tower is only a courtesy so that it can reclaim disk space: a closable session's channels are all closed, so the local copy no longer protects anything (a breach remedy is impossible once the funding output has been spent by a confirmed close). So:

  • Best-effort local delete: if deleteSessionFromTower fails, we log a warning and still delete the local session copy, so a dead/removed tower can no longer make closable sessions accumulate or block startup.
  • Skip deactivated towers: if the user has deactivated the tower, we skip contacting it entirely and prune the session locally.

Testing

Two new cases in TestClient:

  • a closable session is deleted even when the tower is unreachable;
  • a closable session of a deactivated tower is deleted without contacting the still-reachable tower (asserted by the tower server still holding the session).

The full watchtower/wtclient suite passes.

Not in this PR (possible follow-ups)

  • A --force/purge escape hatch so an operator can immediately remove a dead tower and its remaining sessions (relates to [feature]: cleanup of wtclient.db / removal of watchtower #7035).
  • Bounding the per-dial timeout for the cleanup path (currently up to ConnectionTimeout per address). With the two changes here a dead tower's sessions are pruned after a single failed dial rather than re-dialed forever, so this is an optional optimization.

Loading
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

severity-high Requires knowledgeable engineer review

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[bug]: [wtclient] Tower that died make lnd start for hours

3 participants